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instruction  system  to  Chanute.  The  terminals  are  supporting  a  service 
test  to  see  just  how  computer -assisted  instruction  might  oe  used  in 
teaching  a  technical  training  course.  The  project  is  monitored  at 
Chanute  by  the  Training  Research  Applications  Branch.  Because  branch 
personnel  nad  been  interested  in  the  problem  of  the  reading  difficulty 
of  training  literature  for  several  years,  it  was  proposed  that  PLATO  IV 
be  investigated  for  possibilities  for  automating  the  determination  of 
reading  grade  levels  of  training  materials.  The  remainder  of  this  paper 
will  discuss  the  results  of  this  investigation. 

SECTION  C  -  METHOD 

6.  In  deciding  the  ground  rules  for  programming  lesson  PILL  In  the 
PLATO  system  to  determine  reading  difficulty  levels,  the  experiences 
with  past  systems  were  used  as  the  oasis  for  the  program  parameters. 

a.  The  PL4T0  IV  terminal  uses  a  keyboard  similar  to  thar  cf  a 
regular  typewriter.  The  first  ground  rule  set  up  was  that  the  reada¬ 
bility  determination  system  should  require  no  variations  from  normal 
typing  procedure.  This  ground  rule  has  been  modified  to  the  extent  that 
no  periods  are  used  in  a  sample  being  checked  except  to  indicate  the 
end  of  a  sentence.  Abbreviations  and  terms  such  as  etc.,  i.e,,  and  e.g. 
are  typed  intc  the  system  without  the  period. 

o.  Another  rule  adopted  was  that  numbers  would  not  oe  typed  into 
the  system;  e.g.,  $1,000,000. 

c.  The  third  ground  rule  establisned  was  that  the  PIRL  readout 
should  be  in  grade  level  rather  than  on  soon  other  scale.  In  previous 
experiments,  we  have  found  that  our  technical  writers  (as  well  as 
training  specialists  and  supervisors)  can  interpret  a  grade  level,  but 
do  not  relate  well  to  a  numerical  scale. 

7.  In  developing  a  formula  for  the  PLATO  system  to  usl  to  determine 
graae  level,  the  first  consideration  was  to  establish  that  the  factors 
used  relate  to  those  found  in  other  reading  level  indices.  The  element 
relating  to  sentence  length  (words  per  sentence)  is  identical  to  tr.at 
founa  in  most  currently  used  indices  suen  as  the  Flesch  and  Fry  Formulas 
sc,  no  further  verification  for  using  this  component  in  the  PIRL  formula 
was  considered  necessary.  The  verification  of  the  relationship  within 
the  word  structure  was  also  virtually  self-evident  since  an  association 
does  exist  between  the  number  of  letters  in  a  word  and  the  number  of 
syllables  it  contains.  Also,  the  average  number  of  letters  per  word 
apparently  bears  a  fairly  close  relationship  to  the  proportion  of  words 
included  in  a  list  of  most  common  words,  which  is  the  oasis  of  the  Dale- 
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Cha.l  readability  formula.  Consequently,  the  two  factors  used  in  the 
PIR1  formula  (average  number  of  letters  per  word  and  average  number  of 
woras  per  sentence)  should  provide  a  readability  index  va^id  with  ether 
systems  if  proper  weightings  for  each  factor  are  determined. 

a.  To  obtain  a  grade  level  reading,  samples  t  be  measured  are 
selected  and  the  material  is  typed  into  the  PLATO  wesson  called  PI  PL . 
Derailed  instructions  on  how  to  use  th.es  lessen  am  included  in  the 
program.  However,  these  are  not  call:  a  up  unless  the  user  indicates  he* 
n  *vds  them  by  pusning  the  MHelp,?  key.  The  first  sore  n  display  seen 
signing  in*  PIRL  is  shown  in  Figure  1.  Aftvr  a  sample  has  beer. 
cyp^d  into  the*  t  IRL  lesson,  the  "Next1*  key  is  pushed.  The  computer  *her. 
negins  inspect inr  the  words  and  assigns  a  point  value  to  each  word  ar 
follows : 


1,  1  or  7  letters . 1  p  in4* 

u,  5  or  6  letters . 2  points 

7,  6  or  9  letters . 3  points 

10  or  more  letters . Jj  points 

b.  Following  the  word  count,  th*  or.o..ter  c  u.nts  the  number  of 
?mt le te  thoughts/sc r tences  as  indie  v*  d  by  a  period,  question  mark, 

*  >:  lamation  mark,  co.cn,  or  semi-col  -n .  The  total  number  cf  word  points 
is  then  di vised  v;r  number  of  sen^oces .  This  answer  is  then  daviaed 
*hree,  which  is  t.v*n  displayed  on  ~;.s.  *>crer  n  as  the  FT PL,  which  is 
..floated  as  a  grade  .evel .  An  illustration  of  how  this  ippears  on  the 
screen  may  be  seen  in  Figure  2. 

If  tn  rSnif*  Next"  keys  are  pr*.  ss>  d,  the  material  on  the  serve:: 
.o  erased  and  a  new  sample  may  be  typ*  «j  into  the  system.  However,  if 
a  sample  is  typed  int  c  the  system  and  tne  typist  wishes  to  save  tne 
information  concerning  the  sample ,  the  "Data’’  key  is  pressed.  A  data 
ct flection  file  :s  th-n  started  and  information  on  this  first  sample  then 
appears  on  the  screen.  When  the  "Next"  Key  ir  press*  d,  t.ils  data  is 
saved  while  anotner  sample  is  typed  into  *ho  system.  Data  dor  up  *. o 
l-  samples  may  b*.  he  I  n  in  this  way,  with  the  mean  of  all  cm-  samples 
automatically  cal d  whenever  a  new  sample  is  added.  A  data 
collection  file  screen  display  nay  be  seen  in  Figure  3*  This  file 
:.v~  only  shows  the  grade  level  of  each  sample  and  the  mean  of  the 
:  cv.  les,  out  also  shows  tne  sentence  coun*,  total  word  count,  and  the 
uumors  of  words  with  1,  ° ,  or  3  letters;  a ,  S,  or  6  letters;  9  8,  or 
letters;  and  10  or  more  letters. 

d.  The  appropriateness  of  tne  weightings  giver,  to  word  lengths  was 
cstablishea  by  correlating  grade  levels  calculated  by  PIRL  with  grade 
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Figure  1.  First.  »£.spls.y  Seen  Whon 
oigninj  Inio  Lesson  PIRL 


TYPE.  MATLkIAL  AT  Th£  ARROW:  Press  -NEXT-  when  finishes. 


>  This  report  discusses  an  experiment  in  •'»  ;ch  trie  c  <-*•» 

uter  based  instructional  system  i.-ia .  programmed  to  rneas.  *  e  the 
reading  (grace!  levels  of  written  training  material:  T’.-- 

1  esson  m  the  PLATO  system  that  perform*  this  task  is  .-.ie;: 

PIRL,  which  stands  for  PLATO 


Indicated  Reading  Level  .  In  this  Air  r:-r ce  e/r-c-nrer, t,  sai-w 
s  of  technical  training  materials  fry.  five  d;  f i e-ent  cs^ee" 
fields  were  checked  for  their  grade  i  evel  :  te:hr  i  ;a;  wr 

ers  using  the  fog  count  system.  T^e  same  tamc.ee  were  then 
ecked  bo  nine  educat i on/t raining  specialists 


using  the  PI  each  system.  The  samples  were  then  typed  m  the 
LATO  IV  system  by  six  clerk  typists  and  then  ty  seven  PL AT0 
authors.  An  analysis  of  variance  of  the  al i-samcle  mean*  ; 
the  four  groups  indicates  nc  statistical  significant  d.fferen 
e  among  them.  It  is.  concluded  that 


!  the  PLA,-"0  iv 
determining 
tenals  as  a 


lessen  RIPl  is  an  eaeiis’  used 
ana  e  ip-res  a ;  r,g  the  difficult'’ 
graoe  le-.-ei. 


Pea  d  i  ng-  1  eve  1  .  .  .  i  t  . 

Number  cf  sentences . 7  Total  nurruoe:  ;  .  yds...  15$ 

-SHIFT  NEXT-  for  r.e-'  sample.  -DATA  to  st  '  e  this  data 

Time...  7.7  m 1  no t  es 


Figure  2 


Screen  Display  of  Sanple 
and  Its  Data 
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levels  of  the  same  samples  establi ci  by  other  systems.  Thus,  the 
FIRL  formula,  like  most  otner  reucn-il: ty  formulas,  is  derivea  fro r. 
ratios  representing  word  difficulty  (number  of  letters  per  word)  and 
sent  or..';  difficulty  (number  of  words  p<^  r  sentence). 

t.  7c  octiir.  a  ne^at  i  ve^y  broad  base  :'or  verifying  the  FIHL  formula, 
■Initial  v::r>  /a.  cone  with  five,  u*uT0  wor :  samples.  Grade  levels  were 
ie t ermine c  b  -  t:.  Flesch  count ,  the  fog  count,  and  by  PIP  1  and  found  t: 
r.av^  a  high  .  gr<  •  of  correlation.  Then,  *c  verify  t.oe  ;  IKL  lesson  in 
an  operational  s::uati;n,  the  following  experimental  design  was 
y©  2.  ^  n c 

,  a.  * -. **  wore  sample o 

r.ic*^  tram; ..g  materials  were 
tiiffer*'*nt  ca: ~  *  r  field;  i.e., 
engines,  and  automotive. 

o .  Twenty-:?-.- y*  n  technician/authors  from  the  five  training  aesurt- 
r.err :  were  nek  to  perform  a  fog  count  of  the  samples  ana  re  c  rn  t.oeir 
fir:  dir.ru .  how  v*-r ,  nc  technician  performed  a  far  c  ount  on  mat*  rl .  - 
original,  d  -n  nis  department.  As  it  turned  out,  there  were  au'.ncr/ 
technicians  wno  .da  a  fog  count  on  ea^h  sample. 

.-■-typists,  unf  ami  liar  with  the  PLAIT  oyster. ,  v  : 

five  samples  into  the  FLAX  .vrer  and  to  det-rrmn- 
ride  l'_v-r  1  of  each  sample. 

au  h;rs  were  asked  to  cype  t.ie  fivv  sample:  . r.  ' 
to  determine  and  cord  tn-  grade  level  of  tarn 


c  1  raHCvt  paragraphs  from  ^x.stir.v  teen- 
selected.  Each  sample  was  from  an  entirely 
weather,  missiles,  prieudmuilcs ,  ,'e  . 


•  .  Nine  rd.::r  im:  and  Training  Specialists  with  curricula  experience 
.-■"‘re  asreet  *a  perform  a  Flesch  count  of  the  five  samples.  This  :;ut; 
a  is  then  converted  tc-  its  grade  level  using  the  table  in  Fuesch’s  booK, 
rlcw  to  Tes t  Rea dabili t v . 


SECTION  D  -  RESULTS  AND  DISCUSSION 

c  •  T n--  results  achieved  by  the  four  groups  of  people  used  in  this 
study  are  shown  in  Tac^.-s  1  thru 

a.  Fog  count  results  are  snown  in  Table  1.  Differences  in  th-' 
grade  levels  obtained  by  different  writers  can  be  easily  seen  in  the 
range  for  each  sample. 
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Sar.tle 


jraie  ^eve. 

Range 


-^1  .  ^ 


12  .  1-ISr  .  5  =  7  • 


'■lean  Grade 
Level 

13  •  c 


-00 


standard 

Deviation 

1.0 

1.? 

1.7 


lesser. 

anieai 


DIHL 

:rocc 


Taole  1.  Results  of  the  Manual  Fog  leant  of 

Five  Samples  by  21  Technician/Authors. 

.r.nicated  reauing  levels  as  determined  by  the  typists  us-:*g 
snown  in  Table  2.  Because  typing  is  oa si tally  a  mech- 
iure ,  one  wou.d  expect  a  very  high  correlation  :f  the  typists’ 


Sarxle 


Grade  Level 
Range 


Mean  Grade 
Level 


Stan  :xa  ~  c 
Devi a*  .or. 


.  7  -i  v 


1.1 


u- 


'  -la .  6  -  . 
Table-  2 . 


rL suits  cf  FIRL  Grade  Lev  1  Dv  termi.na* 
of  Five  Samples  by  Six  ClerK/Typists. 


*  j  - 


■f  1  taxing  ^ev^is.  However,  th-^  PIHL  method  doe^  cepe  .no 
:uru-:y,  and  with  the  relatively  small  samples  us- a  ir  * v- 


^xperirmc ,  typing  •  rr:rs  did  cause  noticeable  grade  leve..  variations. 
The  typing  •:rror.%.,  in  most  cases,  were  probably  caused  oy  the  different 
touch  of  thv  P;  AT<.  terminal  keyboard. 

c.  Results  a -reeved  by  the  PLATO  authors  using  lesson  PIRL  arc- 
shown  in  Table  7. 
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t  he  study  no cause  <.f  their  backer  June  ana  experience  in  curricula, 

'heir  educational  level,  and  the  fact  they  should  be  more  skilled  a* 
i dent liyin.-T  syllables,  which  is  tne  near:  of  the  Flesch  system.  Even 
higher  level  people  are  not  infallible,  and  m  one  uOO-wcrd  sample, 
there  was  a  count  variation  of  ^0  syllables  between  the  hi ghest  ana 
lowest  counts.  It  is  also  important  that  or**-  be  aware  hat  tne  Fie  sen 
syster.  has  a  range  rather  than  on**  value  to  convert  to  Trade  levels. 

For  example,  a  Flesch  raw  score  of  u2-L6  approximates  a  grad:  level  cf 
la.  The  range  from  u7-51  is  a  grade  level  of  approximately  1;.  Tne 
range  cf  52-57  is  approximately  11-5-  It  is  this  rar.g*  "actor  that 
accounts  for  the  fact  that  for  sample  IV  every  person  using  the  F laser, 
system  car/  up  v;ita  a  grad-  Lgvcl  of  11.3  i>-r  the  sample,  evf  r.  though 
;here  were  variations  in  their  syllable  counts. 

IV.  Possibly  the  most  significant  taolc  for  validating  :nc  PIRL  formula, 
however,  is  Table  5 .  Here  is  shown  the  mean  of  all  th<.  five  samples 
aided  together  for  each  method.  In  other  words,  this  is  the  mean  graefc 

Grade  Level  for  21  TechnicianATriters  12.53 

Using  the  "fog  count”  method 

Mean  Grade  Level  for  Six  Clerk/Typis tc  1^.61 

Using  the  PLATO  IV  lesson  called  PIRL 

!'Lar.  Grad-  L-vel  for  Seven  PLATO  IV  Authors  12.  o 

Using  tne  lesson  called  PIRL 

Meat.  Grade  i.,evei  for  Nine  Education  and  i_./5 

Training  Specialists  using  tne  "Flesch”  count 

Table  5 *  Mean  of  Gracie  Levels  for  the  r-ve  Test 
Samples  Treated  as  One,  Large  Sample. 

xeVt.  of  an  •*  :ui valent  i,030  word  sample.  As  can  be  seen,  the  .means 
r  all  me  th  ^  is  used  tc  determine  grade  levels  are  quite  close.  An 
analysis  of  * h*  variance  of  the  grade  levels  found  by  th*  four  ^ro'dps 
is  shown  in  Tunic  6,  and  indicates  that  there  is  no  statistically  sig¬ 
nificant  difference  in  the  grade  level  determinations .  All  computations 
f :  r  this  study  wvr*  performed  using  statistical  packages  available  in 
Vv  FLA?.  IV  system. 

r  oil  d.f.  3,39  ?  =  O.cioy 

Taole  6.  Analysis  of  Variance  of  tne  Grade  Levels 
of  2,000  Word  Samples  oy  Four  Groups . 
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—  exact  iigurt  r  went;  not  always  ra.pt  on  :r> 
the  i-ffert nt  pr- op A..  to  do  their  gram  iev-i  m 
mlf-way  mrougr.  me  experir.: tAmng  sycmn 
PIRL.  low,  when  tne  PIHL  reaaou:  is  given,  a  r 
of  *lm  i*  took  t:  typ-  m  the  sample  will  air: 
screen  v -"f*  rigur-.  ^  } .  Both  estimates  and  arm 
in  this  e-xpe rim-:/  took  to  determine  tne  read an 
samples  were  useo  to  arrive  at  the  following  ro 


r.  u:r  cf  time  it  took 
mrmnatir  .  About 
w*. .  t r  o gr a.Tt  d  i r. t  c 

x  r  it  on  the  amount 
a  a  *■  e  a  r  o  n  t  he  terminal 
im*.  •  s  t  ne  pa  r  1 1?  ipant  s 
g  irvds  o:'  the  five 
mean  figures: 


PLAT,  at ors  using  PIRL 
Pies  on  counters 


r;  .ours 

.•  minutes 
\  uturs 


Sm  .  rON  u  “  CONCLUSIONS  AN . >  HCOMI 


m.  I*  mould  be  xept  in  nine!  mat  therv 
readability  whirr*  formulas  cannot  measure  . 
sucp  e r *  ma t te r ,  word  order ,  ana  met  c r . ran 
not  nort* )  important  than  word  and  s-  rmonm 
readability.  In  addition,  formula  scores 
errors  in  sampling  or  in  their  ap:a. rat  1 t n 
i  vr on  won  ant  s*-  nt  ^  me  wer..*"  a  .  r  r 


izat^ 


of  the  matt 
cv  a  need  : 


xp*.  r m  r.m  nas  cncvn  rencmg  ap “it  u at  *  may 
stuck?:;*  as  a  high  schorl  graduate,  am  nas  : 
Awl  stores.  Tnus ,  readability  ratings  ear. 
as  a  tool  t:  he  Id  make  one’s  writing  r^ar 


:x 

be 


anc 

ana 


-  w  r  .  -  :  ne  •  o  r s  . -c  nee  r  ni ng 
h  *  ..:*gs  as  imagery, 

•  ::  -v  also  quite  (if 
m  in  oetr r mining 
■  _  r.  a  ccura*  •.  due  t.  o 

w»  v-  r,  dif;A  ruity  It  vein 
.  r  in  uic  at  ore  if 

ai  .  m,  -an  uimt  a 
‘  r  *v  s  ta  *  »“•  w  - 
ng  .a — >  anoerst  ooo. 
i..  writing  wner?.- 
a*f.  :  u:w  t  von  when  the 
r.  naniem  .or  electronic 
m  serve  a  useful  purpose 
more  unde rs tandable . 


; n*  :  LA  It  IV  system  app.  arr 
r'  .r  it; +  e m ning  and  expressing 
.  t  a  gr  a  a*  ie  v  1 1 .  The  ix  a  r  *  n ,  -  w 
:  ram; tr,  Lowry  anc  Shepparu. 

partm-nto  and  technical  writers 
;  ust  lesson  PIRL  to  cm  ok  n<-  r 


wi * ♦ 


f  mm  study,  the 
a : .  ■*  a r i 1  y  used  and 
r  ■  *  tiL  —xj.  y  w- 1  written 
'  .  *  .  rmma^s  at  three 

- iv  t  omne  nde  a  t  nat 
me  oases  take  advantage 


!•'  VviS 


tneir  training 


1 1 
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